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(54) Measurement-based management method for packet communication networks 



(57) Disclosed are network management proce- 
dures that apply measurements of traffic load to achieve 
greater efficiency in the operation of the network. In a 
method for deciding whether to route an incoming call 
on a selected potential service route, the potential serv- 



ice route is treated preferentially if each of its links has 
available capacity that is more than sufficient by a spec- 
ified margin. In a method for computing billing revenues, 
the non-compliance of the network service provider with 
contracted requirements for carried load causes a rev- 
enue penalty to be exacted for lost bandwidth. 
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Description 

Field of the Invention 

[0001] The invention relates to methods of admission control, routing, and billing computation in packet networks. 
More particularly, the invention relates to the use of such methods in operating networks that support multiple levels 
of service. 

Art Background 

[0002] A Service-Level Agreement (SLA) is a contract between the provider of packet network services and one of 
its customers (i.e., subscribers) that specifies some minimum quality of service that must be met or exceeded in the 
handling of calls identified with an application or service. One measure of performance that an SLA may specify is the 
amount of bandwidth that must be available on demand. A Virtual Private Network (VPN) is defined when the SLA 
specifies the amount of bandwidth that is to be made available, on demand, in each of a set of streams identified with 
the customer. A "stream" in this context is a node pair o of the network consisting of a source node and a destination 
node with respect to calls, in association with a particular class of service s. The various possible service classes may 
include, e.g., voice, data, e-mail, file tranfers, web browsing, and video. A packet network is, for example, a network 
supporting the ATM, IP, or Frame Relay protocol. 

[0003] We will use the term "call" to denote any communicative transaction , or distinct subdivision of a communicative 
transaction, commonly referred to as a call, connection, or flow. 

[0004] In the operation of a packet network, incoming calls identified with various customers must compete for the 
same network resources, such as link bandwidth capacity. Additionally, there is contention for the same resources by 
calls of different service classes, whether belonging to the same customer or to different customers. In such an envi- 
ronment, it is difficult to consistently provide each customer with the service quality it demands in each class of service, 
while also profitably operating the network. 

[0005] One approach to this problem is to design the bandwidth loads of the network to accommodate the ex- 
pected traffic patterns in an optimal way. Here, the design load X sr is the designed bandwidth to be carried on a service 
route (s, /), i.e., on a route /-between a given source-destination pair in a given service class s. A design method that 
explicitly recognizes the statistical properties of communication traffic is described, e.g., in U.S. Patent No. 5,854,903 
issued to D. Mitra et al. on Dec. 29, 1 998 under the title "Optimization Method for Routing and Logical Network Design 
in Multi-Service Networks" and commonly assigned herewith. An extension of that exemplary design method to virtual 
private networks is described in EP-A-0952741 . 

[0006] A design method based on concepts relating to mufticommodity flow is described in EP Application No* 00 
306 514.1. 

[0007] Although such off-line methods of network design are useful, they do not, by themselves, provide the ability 
to respond to traffic behavior in real time. However, because of the randomly fluctuating nature of traffic, there are often 
potential gains in total carried traffic or total revenue that could be realized if routing decisions - could be informed by 
real-time measurements. 

Summary of the Invention 

[0008] We have developed procedures for network management that apply measurements of traffic load (i.e., of 
traffic bandwidth) in order to achieve greater efficiency in the operation of the network. 

[0009] In one aspect, our invention involves a method for deciding whether to route an incoming call on a selected 
potential service route. According to our method, the potential service route is classified as oversubscribed or under- 
subscribed, depending on how its measured load compares with its design load. A potential service route that is over- 
subscribed will be deemed to have sufficient available bandwidth capacity to carry the incoming call only if each of its 
links has available capacity that is more than sufficient by a margin referred to here as the bandwidth reservation. 
[0010] In another aspect, our invention involves a method for computing billing revenues, in which incremental rev- 
enues for a given stream depend on whether the network service provider is deemed compliant with an SLA with 
respect to the given stream. According to one embodiment, the service provider, to be deemed compliant, must carry 
at least a contracted fraction of offered load (i.e., of offered stream bandwidth) when the offered load lies within a 
contracted limit, but need only carry a specified load when the offered load exceeds the contracted limit. A revenue 
penalty is exacted for offered stream bandwidth that is lost while the service provider is non-compliant. 
[001 1] In yet another aspect, our invention involves performing all of the following steps at the ingress node for an 
incoming call destined for a given stream: determining whether each of at least some potential service routes for the 
incoming call is oversubscribed or undersubscribed; from measurements of offered and earned load, determining 
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whether the network is compliant with a condition, e.g., the SLA condition described above; routing the incoming call 
according to a procedure that preferentially selects undersubscribed service routes; and accruing a positive revenue 
increment in at least one time window for carried load due to the routed call. 

Brief Description of the Drawing 

[0012] 

FIG. 1 is an annotated flowchart which illustrates compliance decisions in accordance with an illustrative type of 
SLA. As illustrated, four outcomes are possible, depending on: (i) whether or not the customer complies with 
contracted limits on offered load in a given stream, and (ii) whether or not the service provider (the "SP" as indicated 
in the figure) carries a contracted amount of the offered load. 

FIG. 2 is a flowchart illustrating incremental stream revenue calculations in accordance with an illustrative embod- 
iment of the invention, in one aspect. At the final summing point near the bottom of the figure, a positive revenue 
increment for carried load and a n egative penalty i ncrement for lost load are combined to form a net stream revenue 
increment. 

FIG. 3 is a flowchart illustrating determination of the loading status of a service route by comparting measured 
load with design load, according to an illustrative embodiment of the invention, in one aspect. 
FIG. 4 is a flowchart illustrating bandwidth protection in routing decisions, according to an illustrative embodiment 
of the invention, in one aspect. 

FIG. 5 is a flowchart illustrating the handling of a request for a new call for a given stream according to the invention 
in one embodiment. Included in the procedure of FIG. 5 is the bandwidth protection procedure of FIG. 4. 
FIG. 6 is a chart illustrating an exemplary form of bandwidth protection that may apply when VPNs are supported 
by the network. 

FIG. 7 is a schematic diagram of a fictitious communication network used as the basis for numerical studies de- 
scribed in the Example section, below. 

Detailed Description 

[0013] An exemplary SLA pertinent to the practice of the invention stipulates, for each stream (s,o), an aggregate 
offered bandwidth U OT (the "contracted offered bandwidth") and an aggregate carried bandwidth ^(the "contracted 
carried bandwidth"), < U^. Implicitly, the ratio / is the contracted flow-acceptance ratio for the stream. It 
should be noted that this ratio cannot be precisely unity, because due to the statistical nature of the incoming traffic, 
only a network having infinite capacity could guarantee that 100% of incoming calls will be accepted. 
[0014] For determining whether there is compliance with the terms of the SLA, estimates of the actual offered and 
carried bandwidths are made, based on measurements. Bandwidth can be measured directly by examining the offered 
and carried packets. Alternatively, calls can be counted and the total bandwidth inferred from effective bandwidths 
associated with each of the calls. (Effective bandwidth is described in more detail below.) In either case, it is advanta- 
geous for the bandwidth measurements to be performed at the ingress node, i.e., at the source node of the corre- 
sponding stream. 

[0015] Initially, we will describe an SLA monitoring scheme based on call-level accounting. Later, we will discuss an 
example of SLA monitoring based on packet-level (i.e., on data-level) accounting. The numerical studies that we de- 
scribe below used call-level accounting. 

[001 6] An exemplary measurement procedure employs time windows, referred to here as "SLA windows," and it also 
employs exponential smoothing. The SLA window length t and the smoothing parameter a SiA are also advantageously 
stipulated in the SLA. 

[001 7] Let?, (n) denote a measured value of carried stream bandwidth in time window n, and let*. (n)denote a meas- 
ured value of offered stream bandwidth in the same time window. Because each measurement involves some degree 
of estimation, we refer to these values as "estimated" bandwidth values in the following discussion. 
[0018] In the following discussion, it will be optional whether smoothed or unsmoothed values of*. {n)an6a m (n) are 
used. (Smoothed values were used in the numerical studies described below.) However, to illustrate one form of smooth- 
ing that is useful in this context, we here let? (n) and<E (n) represent smoothed values, and we let^r (n) and*- (n) 
represent corresponding raw, i.e., unsmoothed, values. Then according to an illustrative smoothing technique, 



V™ <»+!) = a^V? (») + (1 - ar^ )V™ (*) . 
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[0019] According to an exemplary SLA, a compliant customer is one whose offered stream bandwidth does not 
exceed the contracted amount. The service provider promises to carry the same fraction of the estimated offered 
bandwidth as the proportion of contracted carried to contracted offered bandwidth. If the service provider carries a 
smaller fraction than what is promised, ft is declared non-compliant and pays a penalty for each call that is lost (i.e., 
not earned) while the service provider is in the non-compliant state. 

[0020] On the other hand, the customer is non-compliant if it offers more stream bandwidth than the contracted 
amount. In that event; the service provider promises to carry only the contracted amount of carried bandwidth. The 
service provider is declared non-compliant if it fails to cany the contracted amount. In that case it pays a penalty for 
lost calls, e.g. for lost bandwidth up to the contracted amount. 

[0021] Advantageously, the monitoring of customer and service-provider compliance, and the declaration of corre- 
sponding compliant and non-compliant states, take place at the ingress node. 

[0022] FIG. 1 illustrates an exemplary decision process for SLA compliance. At block 5, the estimated valuer (n) 
of the offered stream bandwidth is compared with the contracted value U^. The estimated value of the offered band- 
width (and as will be seen below, also the estimated value of the carried bandwidth) is determined at the end of the 
rrth SLA window. However the variable "SLA_state", which takes the value "compliant" if the service provider is SLA- 
compliant and the value "non-compliant" otherwise, is treated as uniform over the entire window. (More generally, the 
pair of variables describing the respective states of SLA compliance of the customer and the service provider are 
treated as uniform over the entire window.) We found that this approximation is helpful for controlling the processing 
burden, and that it permits averaging and tends to increase accuracy. 

[0023] The output of block 5 is "yes" if the estimated value of offered bandwidth is no greater than the contracted 
value. In that case, the customer is SLA compliant, as represented by the left-hand side of the grid at the bottom of 
the figure, i.e., quadrants A and B. If the output of blocks is "no", the customer is SLA non-compliant, as represented 
by quadrants C and D. 

[0024] The test of whether the service provider is SLA-compliant (which, in turn, determines the value of the variable 
SLA_state) takes different forms, depending on the result of block 5. In the case of a compliant customer, the test of 
block 10 applies. In block 10, ratios are compared of carried bandwidth to offered bandwidth. If the ratio / of 
contracted values is no greater than the ration (n)lo m (n) of estimated values, the service provider is declared SLA- 
compliant for window n, as indicated in quadrant A of the figure. Otherwise, the service provider is declared SLA non- 
compliant, as indicated in quadrant B. 

[0025] In the case of a non-compliant customer, the test of block 15 applies. According to the test of block 15, the 
service provider is declared SLA-compliant for window n if the contracted amount of carried bandwidth is no greater 
than the estimated amount*^/)), as indicated in quadrant C of the figure. Otherwise, the service provider is declared 
non-compliant, as indicated in quadrant D. 

[0026] Every call that is carried generates revenue to the service provider and increments a flow revenue measure 
w so( n )- For example, as shown in FIG. 2, a previous cumulative revenue measure IV^n-l ) (shown in block 20 of the 
figure) is incremented at summing point 30 by the current amount shown in block 25 to form the current cumulative 
revenue measure W^n) for SLA window n (block 35). The current increment of block 25 is the product of the number 
of caHs of stream (s,o) carried in window n, and a stream revenue parameter By way of example, but not 
of limitation, we note that in numerical studies we set equal to the product of the effective bandwidth d s and the 
mean holding time h s of calls of service class s. The effective bandwidth can be adjusted to account, in a single pa- 
rameter, for various packet-level factors such as burstiness, delay, jitter, and loss at network elements. 
[0027] If the service provider loses calls white in a state of SLA non-compliance, it may be liable to pay a penalty. In 
the exemplary scheme of FIG. 2, a previous value penalty ^(n -1) of a cumulative flow penalty measure (block 55) is 
incremented at summing point 60 by the current penalty increment to form a current value penalty^n) of the cumulative 
measure for window n (block 65). The current penalty increment is the value shown at block 40 of the figure. However, 
at multiplier 50, the current penalty increment is given a multiplicative weight of 0 (in which case it is not added to the 
cumulative value in block 65) if the service provider is SLA-compliant in window n. Otherwise, the penalty increment 
receives a multiplicative weight of 1 . 

[0028] As shown at block 40, the penalty increment is exemplarify the product of three factors: the stream revenue 
parameter the number N^n) of calls of stream (s,a) that are lost in SLA window n, and an adjustable penalty 
multiplier m^, which is typically greater than 1 . 

[0029] Various alternative penalty structures are also readily implemented. For example, the penalty structure of 
FIG. 2 penalizes the service provider for all calls that are lost while the SLA state of the network lies in quadrant D of 
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FIG. 1, even when the amount of offered bandwidth is grossly in excess of that stipulated in the SLA. To discourage 
gross excesses of offered traffic, it will in some cases be advantageous to limit the factor in block 40 of FIG. 2 
so that it includes only the difference between the measured (i.e., estimated) and contracted values of earned band- 
width. 

5 [0030] At summing point 70, the cumulative stream revenue value of block 35 and the cumulative stream penalty 
value of block 65 are combined as respective positive and negative contributions to the net stream revenue W^et^ 
(n), as shown at block 75. Summing W^net^n) over all streams gives a network-wide measure W_net{n) of cumulative 
net revenue, as shown in block 80. 

[0031] In the preceding discussion, we have treated it as optional whether smoothed or unsmoothed values are used 
10 for? w (n) and*, (n). According to our current belief, however, it will be especially advantageous to base the SLA state 
determination on smoothed values, but to compute the revenue and penalty values based on the unsmoothed meas- 
urements of bandwidth offered and carried in each time window. 

[0032] As mentioned above, an alternative to call-level monitoring is to measure the offered and earned bandwidth 
at the packet (or data) level. Leaky bucket techniques, for example, are readily used to perform such measurements. 
15 (Leaky bucket measurements will tell how much bandwidth was carried and how much was dropped or marked as 
non-compliant. Thus, the amount offered is readily inferred.) In the context of packet-level measurements, we let oo^ 
represent the revenue generated by the service provider for carrying a unit amount of data on stream (s, o). Thus, an 
expression appropriate in this context for the incremental gain in revenue for window n is 



[0033] A penalty structure that we believe will be especially advantageous in the context of packet-level measure- 
ments is defined by prescriptions (i)-(i»), below, for the value of the incremental penalty for time window n, i.e., for 
25 penatty^inypenalty^^ ). The prescription are made with reference to quadrants A-D of FIG. 1 . 

(i) If the network SLA state for stream (s,o) lies in quadrant A or C, the incremental penalty is zero. 

(ii) If the SLA state lies in quadrant B, the incremental penalty is 



30 



35 



I if- PTM-vS"m 



(iii) If the SLA state lies in quadrant D, the incremental penalty is 



[0034] The notation [».J* signifies that if the bracketed quantity is less than zero, it should be set to zero. 
[0035] As noted, an off-line design process is advantageously employed for allocating (in a statistical sense) the 
offered traffic for each stream among the admissible routes for that stream. Information passed from the design phase 
to the SLA-management process will generally include and as well as the designed service-route loads X^ 
We have found ft advantageous to derive the loads X^from the raw output of the design, which is based on mean 
values of traffic bandwidth, in a manner which reserves extra capacity in anticipation of traffic variability. Thus, if the 
design process yields a mean value of aggregate bandwidth carried on a service route, we set the corresponding 
load parameter X^ equal to M^plus an additional increment related to traffic variability. Although the standard deviation , 
for example, could be used as such a measure of variability, we have found that an adequate measure is provided by 
the square root of the mean value. Accordingly, we have found it advantageous to set 



55 X sr= M sr + iffilr 

where y is a small non-negatfve number, typically about 0.5. Similarly, we have found it advantageous to set 



5 
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where p is another small, non-negative number, also typically about 0.5. In the preceding expression, is the mean 
5 carried aggregate bandwidth on stream (s,a) obtained from the design process. As p increases, the contracted amount 
v so of canned bandwidth decreases more steeply with increasing traffic variability. Thus, increasing p is appropriate 
for reflecting increasing aversion by the service provider to incurring penalties for lost calls. On the other hand, increas- 
ing p also tends to reduce the flow-acceptance ratio I contracted for in the SLA. 

[0036] A penalty structure for lost calls, as described above, can optionally be included in the design process, al- 
io though some additional complexity will result. In the numerical studies whose results we report below, we did not include 
the penalty structure in the design process. 

[0037] Alternate revenue structures are also readily implemented. For example, the service provider might wish to 
demand a premium for carrying calls at the contracted bandwidth value when the amount of offered bandwidth exceeds 
the contracted value, i.e., when the network state lies in quadrant C of FIG. 1 . In such a case, a second-tier revenue 
is parameter, largerthan the basic stream revenue parameter iv^, can be applied when the network state lies in quadrant 
C. Such a second-tier parameter can be applied, e.g., to all carried bandwidth, or it can be made to apply only to carried 
bandwidth in excess of the contracted amount. 

[0038] In the phase of network management that we refer to as "route classification," each ingress node evaluates, 
for every time window n, a variable status^ (n) based on the bandwidth load, aggregated over calls, of each service 
20 route (s, r) from the ingress node, and it maintains a database of these variables. Each variable status Jn) is computed 
at the beginning of time window n and remains fixed during the window. This status variable is computed for each 
admissible route r for each stream having the given node as its ingress node, for each corresponding egress node, 
and for each service class s. 

[0039] FIG. 3 illustrates an exemplary process for evaluating status^n). At block 85, the measured bandwidth load 
25 z sA n ) on service route (s, r) at the beginning of window n is compared with the design load As indicated at block 
90, the loading status of the service route is declared "undersubscribed" (i.e., status ^n) is set equal to US) if the 
measured load is no greater than the design load. As indicated at block 95, the loading status is declared "oversub- 
scribed" (status^n) is set equal to OS) if the measured load is greater than the design load. The loading status of 
service routes is important in the implementation of the phase referred to here as "Routing and Admission Control," 
30 which is described below. 

[0040] We will now describe an exemplary procedure for measuring the service-route bandwidth load Z sl (n) using 
quantities computed from local measurements at the ingress node. This measurement procedure is based on a window 
of length t and on exponential smoothing with a smoothing parameter a. A similar procedure, possibly using different 
values of the window length and smoothing parameter, is readily applied for computing the offered and earned stream 
35 loads*, (n) and?, (n). 

[0041 ] Let t represent a time value with in the nth window, i .e., )r5f < mr. Let V^f) denote the aggregate bandwidth 
usage on service route (s, r) at time t. We note that YJfi increments by a unit of the effective bandwidth d s with each 
new call, andjt decrements by the same amount with each call departure. 

[0042] Let Y^n) denote the mean bandwidth usage on the service route over the /rth window, i.e., 



Let ZJ^n + 1 ) denote the exponentially smoothed estimate of bandwidth usage, aggregated over calls, on the service 

route at the start of the {n + 1)th window. . - . 

[0043] Then according to our method, {n + 1 ) = aZ^n) + (1 - a) Y^n). 

[0044] It should be noted in this regard that because only the ingress node will have been setting up calls on service 
route (s, /), without interference from other nodes (which of course may be ingress nodes as to calls for their own 
streams), all the necessary load information is available to iL 

[0045] We now turn to a description of control algorithms for Routing and Admission Control. We note first that these 
algorithms apply a methodology known as Virtual Partitioning (VP). In the VP methodology, the bandwidth capacity of 
each link / is regarded as a resource which is an object of contention by all service routes in which link / is included. In 
our application of VP, those contending service routes that are undersubscribed (at a given time) are given preference 
over oversubscribed service routes. As explained below, a procedure referred to here as Bandwidth Protection (BP) 
implements this preference when new calls associated with a given stream are set up. It should be noted that at call 
set-up, in an exemplary implementation, the ingress node sends to each link in a service route of interest a request 



40 




45 
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and an indication of the value of status^ (n) for the service route of interest and current time window n. 
[0046] We now describe the Bandwidth Protection procedure with reference to FIG. 4. This procedure is advanta- 
geously performed at the ingress node. Let / represent a link traversed by the potential service route, let C, represent 
the bandwidth capacity of link /, and let y/f) represent the total bandwidth usage on the link at the time / of call set-up. 
s (Obviously, y/f) cannot exceed C t ) Let (s, r) represent a potential service route that has been selected for routing an 
incoming call. 

[0047] At block 100 of the figure, a determination is made whether the status of the potential service route in the 
current window n is undersubscribed (i.e., whether status^n) equals US). If the service route is identified as under- 
subscribed, a further determination is made at block 1 05 whether there is sufficient available bandwidth on the service 

io route to accept the call. At block 105, there will be deemed sufficient bandwidth only if at the time of call set-up, for 
every link I traversed by the potential service route, there is enough remaining capacity to accommodate the effective 
bandwidth d s of the incoming call, i.e., only if, for all / E (s, /), y/f) + d s $C t If this condition is satisfied, the call is 
accepted, as indicated at block 115. Otherwise, the call is rejected, as indicated at block 120. 
[0048] With reference once again to block 100, if the service route is determined not to be undersubscribed, it is 

is oversubscribed (i.e., status^n) equals OS). In that case, the determination whether there is sufficient available band- 
width on the service route to accept the call is made at block 110. The test applied at block 110 is more demanding 
than the test applied at block 1 05. At block 1 1 0, each link /traversed by the service route is required to have remaining 
capacity not only for the effective bandwidth d & but also for a q uantity of bandwidth Rd t referred to here as th e bandwidth 
reservation. That is, the call is accepted (at block 125) only if, for all / £ (s t r), y/f)+d s + Rd<> C t Otherwise, the call is 

20 rejected (at block 1 30). 

[0049] The bandwidth reservation Rd forces our routing procedure to give preference to undersubscribed service 
routes in two respects. First, an attempt to route a call on an oversubscribed service route must satisfy a more de- 
manding test than a routing attempt on an undersubscribed service route. Second, enforcing the bandwidth reservation 
assures that after successfully routing a call on an oversubscribed service route, each link along that route will still 
25 have capacity to carry a call on at least one undersubscribed service route in which such link is included. (Depending 
on the value of fl, there may be remaining capacity to carry calls on several undersubscribed service routes.) 
[0050] The banjiwidth reservation described here is the product of two factors: the bandwidth protection parameter 
R and a quantity d. The bandwidth protection parameter is an adjustable, small positive number typically in the range 
1 .0-2.0, and exemplariry about 1 . The quantity oMs, e.g., the greatest effective bandwidth over all service classes; i.e., 

30 

35 [0051] It should be noted that an attempt to set up a call on a selected service route will succeed only if all the links 
in the service route accept the call after the Bandwidth Protection procedure of FIG. 4 has been implemented. 
[0052] As noted, the quantity y/f) represents the total bandwidth usage on a link / at the time t of call set-up. There 
are various ways for the ingress node to acquire this information concerning bandwidth usage on the links belonging 
to pertinent routes. One approach is for the ingress node to send out scout requests as needed, exemplariry by sending 

40 out specialized scout packets, which solicit usage information from the pertinent routers. Such an approach is effective, 
but it contributes a relatively large amount of signalling traffic overhead to the network, which may be disfavored in at 
least some cases. An alternative approach, sometimes referred to as "periodic flooding,'" is for the ingress node to 
broadcast periodic requests to the network, for usage information. This approach adds less traffic overhead than the 
use of scout packets, but late in the broadcast cycle, before the next request, the ingress node is generally forced to 

45 use outdated information. 

[0053] Yet a third approach, which we believe will be advantageous in at least some cases, applies usage information 
that the ingress node has acquired through previous call set-up requests. The advantage of this approach is that it 
adds little or no signaling traffic overhead, and for at least some routes is as current as the most recent routing attempt. 
The use of previous call set-up attempts to acquire link usage information is discussed, e.g., in EP-A-0 777 362. 

so [0054] Turning now to FIG. 5, there is represented at block 1 35 a request to route a new call for stream (s,a). Blocks 
140 and 145 represent an attempt to route the call according to a procedure known as sticky routing. Sticky routing is 
described, e.g., in R.J. Gibbens et al. f "Dynamic alternative routing-modeling and behavior," Proc. 12th Int. Teletraffic 
Congress, Torino, 3.4A.3.1-3.4A.3.7 (1988). 

[0055] The ingress node has the option of attempting to route the new call on any admissible route for the pertinent 
55 stream. According to the sticky routing procedure, the preference is to use the last service route on which a call for the 
same stream was successfully routed. In our exemplary procedure of FIG. 5, however, such a last service route (denoted 
in block 140 as "currents, /)" ) may be selected only if it is undersubscribed in the current time window n. Thus, if the 
test for undersubscribed status of block 1 40 is satisfied, the service route currents, r) is selected for the routing attempt 
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as indicated at block 145. 

[0056] If the test of block 140 is not satisfied, then as indicated at block 150, a determination is made whether, in the 
current time window, there is any service route in the admissible route set R(s,o) that is undersubscribed. If there is at 
least one such service route, a set R^s^n) of the admissible service routes at time window n is defined, and as 
indicated at block 1 55, a member of that set, exemplarily a randomly chosen member, is selected for the routing attempt. 
[0057] If at block 150 no admissible undersubscribed service routes are found, then, as indicated at block 160, a 
preferred one of the available oversubscribed service routes is selected. The preferred oversubscribed service route 
is the one that is determined to be maximally underloaded. In this context, the amount of underloading is the amount 
by which the design load X^ exceeds the aggregate bandwidth usage YJt) on a service route at time t. Thus, the 
maximally underloaded route is the route of the admissible route set that minimizes the quantity YJ.fj-X^ It should be 
noted that the determination of a maximally underloaded route is readily determined at the ingress node, since the 
ingress node has possession of the values of V^f) and 

[0058] Once a service route has been selected, the attempt to set the call up on the selected route is made at block 
1 65, where the Bandwidth Protection procedure of FIG. 5 is implemented. A determination is made at block 1 70 whether 
the routing attempt was successful. If so, then in accordance with sticky routing, if used, the register containing the 
last successful service route current(s, r) is updated, as indicated at block 175. If the test at block 170 indicates an 
unsuccessful attempt, then the call may be lost or, alternatively, a new attempt may be made to route the call according 
to a procedure, described below, that we refer to as crankback. 

[0059] If sticky routing is being applied, then if the test at block 1 70 indicates an unsuccessful routing attempt, current 
(s, r) is set to a null value, as indicated at block 180. 

[0060] When an attempt to set up a call on a selected service route has failed, the likelihood that the service route 
can accept another call set-up request will be small initially, but will increase with time. Accordingly, it will generally be 
advantageous to remove the selected service route from consideration for a period of time 7^ , which we refer to as 
the recovery time. The removal of such a route from the route selection procedure for a period T is indicated in FIG 
5 at block 180. ™ e 
[0061 ] As indicated at block 1 85, monitor data are updated with the results of the call set-up attempt of blocks 1 35-1 80. 
By "monitor data" is meant information to be used in status decisions, revenue and penalty calculations, and the like. 
Such information includes, e.g., entries in databases at the ingress node that keep track of the number of calls carried 
and blocked, the carried and blocked bandwidth, and the like. 

[0062] As noted, if the call set-up attempt has failed, a new set-up attempt may be made by applying a crankback 
procedure. According to an exemplary crankback procedure, after block 185, the procedure of blocks 140-185 is re- 
peated until the new call has been routed, or until the set-up request has failed a specified number of times. In at least 
some cases, it may be advantageous to apply crankback only if certain conditions are satisfied. For example, in one 
form of selective crankback, a new set-up attempt is made only if loss of the call would cause the service provider to 
incur a penalty, i.e., only if the service provider is currently SLA-non-compliant with respect to the relevant stream. 
[0063] We have noted, above, that information passed from the off-line design phase to the SLA management proc- 
ess will generally include the design value of offered stream bandwidth, the design value of carried stream 
bandwidth, and the mean values M^of aggregate bandwidth carried on the respective service routes corresponding 
to each stream. From the values as noted, we obtain designed service-route loads X^ 

[0064] We have also noted, above, that a VPN is defined when the SLA specifies the amount of bandwidth that is 
to be made available, on demand, in each of a set of streams identified by the customer. The concept of SLA compliance 
described above in regard to offered and carried stream bandwidth is readily extended to address compliance issues 
where a VPN has been specified. That is, where previously the tests of blocks 10 and 15 of FIG. 1 were applied to 
quantities V^, (n),*, (n) specific to a given stream (s,o), the same tests are now applied to analogous quantities 

^o' ^ /, )' whicn are specific to a given sub-stream (s, o; v) which belongs to a particular VPN having the 

index v. We refer to such a sub-strearri as a "VPN stream." 

[0065] Thus, a revenue and penalty structure as discussed above in connection with FIG. 2 is readily devised to 
govern the net revenue that the service provider can collect from a customer by virtue of operating a VPN for the 
customer. 

[0066] One or more VPNs may be specified as input to the off-line design phase. In such a case, service-route loads 
X^ that are analogous to the earlier-mentioned loads X^ but are specific to the traffic of particular VPNs v, are ob- 
tainable, directly or after modification, from the design-phase output. We refer to the loads )6 V) as "VPN service-route 
design loads." «" 

[0067] We have noted, above, that various service routes (both for the same stream and for different streams) contend 
for limited bandwidth capacity on those links that are shared among the sevice routes. If too much traffic is routed 
through a given link, a network roadblock can result. Our Bandwidth Protection procedure helps to prevent such road- 
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blocks by reserving link bandwidth on oversubscribed service routes that can be made available to undersubscribed 
service routes intersecting the same links. 

[0068] When VPNs are introduced, additional forms of contention appear. For example, different VPNs may now 
contend for the same link bandwidth, and within a single VPN, different streams as well as different routes belonging 
5 to the same stream may contend for the same link bandwidth. These forms of contention are readily dealt with by a 
simple extension of the Bandwidth Protection procedure of FIG. 4. The earlier concept is extended by defining a new 

variable statu^i")* which is analogous to the above-defined variable status^n), but is specific to a service route 
io belonging to VPN v. A VPN service route (s,r, v) is declared undersubscribed, and statud v) (n) is set equal to US, if the 
measured load 2 (v) (n) on VPN service route (s, r; v) in time window n is no greaterthan the design load . Otherwise 

^ ST ' 

the VPN service route is declared oversubscribed, and statu^ v) ln) is set equal to OS. 

» A 

is [0069] As in the procedure of FIG. 4, a bandwidth reservation R^ d\s imposed if a call is routed on an oversubscribed 
VPN service route. 

[0070] Within a given VPN, there also may be contention between the various classes of service associated with 
that VPN, TTiat is, it will often be the case that the owner of a VPN is less concerned with the call acceptance rate for 
a particular class of service than he is with the cumulative acceptance rate of calls of all classes. Such a VPN owner 

20 will wish to prevent calls of a particular service class to dominate the network resources and crowd out calls of other 
classes. In such an environment, it is useful to characterize a given VPN source-destination pair as oversubscribed if 
it is getting more than its designed share of traffic. A new call, of any service class, will be routed between an over- 
subscribed pair only if a bandwidth reservation rf^d ls imposed on the resulting VPN service route. 
[0071 ] As a general rule, the bandwidth reservation parameter ^ will be common to all VPNs on the n etwork, whereas 

25 the bandwidth reservation parameter fl^ will be separately negotiated for each VPN. Generally, fl< v) will be at least 
as great as R v 2 
[00721 The preceding concepts are described in further detail with reference to FIG. 6. In box 190, the variable 
statud^ (n) takes on the value US or OS, as explained above. A further variable statusf^ (n) is introduced in box 1 95. 
This fdrfher variable is defined with reference to a VPN design load )6 V) obtained by sufnming the load variables )6 V) 

30 over all service classes and all admissible routes. That is/ ° * 



35 



r<»> 



where R M(s, o) is the admissible route set for VPN stream (s,a; v). The design load )6 v) is compared with a measured 

o 

load 2^ {n) equal to the carried bandwidth in time window n for VPN streams & o; v) summed over all service classes. 
If £ v) (n) is no greater than )6 V \ the variable stated (n) is set equal to US. Otherwise, it is set equal to OS 

o o o 

[0073] Quadrant A of the figure represents the state in which both statud^(n) and statud^(n) are equal to US. In 
that case, an incoming call is accepted for routing on the proposed service route without imposing a bandwidth reser- 
vation. 

[0074] Quadrant B of the figure represents the state jn which sfa/us*^(n) is OS, but statud v) (n) is US. In that case, 

the call is accepted only if a bandwidth reservation R^ d is available. * ° 

[0075] Quadrant C of the figure represents the state irnwhich statu^(n) is US, but statud^(n) is OS. In that case, 

the call is accepted only if a bandwidth reservation ft * d is available. " ° 

[0076] Quadrant D of the figure represents the state 2 in which statud v) {n) and statud v) (n) are both OS. In that case, 

the call is accepted onlyjf both of the bandwidth reservations described above are availaBle, i.e., only if a total bandwidth 

reservation (R A + Ff v ^)d\s available. 

[0077] Those skilled in the art will appreciate from the preceding discussion that VPN traffic can be studied at various 
levels of aggregation. At a low level of aggregation, traffic can be studied at the level of VPN service routes, identified 
by the triplet of indices (s,r;v). (It is understood that all the routes r referred to correspond to some given source- 
destination pair a consisting of a source, i.e., ingress, node o A and a destination, i.e., egress, node o 2 .) At a higher 
level, traffic is aggregated over all routes corresponding to a given stream. This defines the VPN stream level, identified 
by the triplet of indices (s, o; v). 
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[0078] At a still higher level, VPN stream traffic is aggregated over all service classes. This defines the VPN pipe 
level, identified by the pair of indices (o; v). It will be appreciated that the variable statu (n), defined above, refers to 
traffic loading at the VPN pipe level. ° 

[0079] At yet a higher level, VPN pipe traffic is aggregated over different source-destination pairs a sharing a common 
ingress node o v In other words, all VPN pipe traffic from a given ingress node is aggregated together. This defines 
the VPN hose level, identified by the pair of indices (o^v). 

[0080] The method described above with reference to FIG. 6 is designed to regulate the sharing of bandwidth by 
VPN service-routes and VPN pipes. In at least some cases, it will be advantageous to apply the method of FIG. 6 at 
a higher or lower level of aggregation than the VPN pipe level. That is, a variable analogous to statv^in) is readify 
devised at the VPN stream level or the VPN hose level, and applied in the method of FIG. 6 in substitution for status 1 ^ (n). 

o 

Example 

[0081] We performed a numerical case study based on a fictitious network which has eight nodes (A/= 8), of which 
10 pairs are directly connected, as shown in Figure 7. The network has 20 directed links (L = 20), one in each direction 
for each connected node pair. The typical bandwidth of a directed link is OC3 = 1 55 Mbps, with the exception of the 
links connecting Argonne (3) and Princeton (4), and also Houston (8) and Atlanta (7), which have bandwidths of 2 x 
OC3 = 310 Mbps. One measure of total resources in the network is 24 OC3-hops. 

[0082] There are six service classes: voice, data 1 , data 2, data 3, data 4, and video, indexed by s = 1 , 2, .... 6, 
respectively. The effective bandwidths of individual flows of these classes are d s = 1 6, 48, 64, 96, 384 and 640 Kbps. 
Voice (s = 1 ) and video (s = 6) are delay sensitive service classes, and their admissible route sets R (s, o) consist only 
of routes with the minimum number of hops. There are a total of 68 routes for each of these two service classes. The 
four remaining are data service classes, all delay insensitive. Their admissible route sets R (s, o), s = 2, 3, 4, 5, are 
identical and consist of routes with at most four hops. For each such s there is a total of 1 60 routes. 
[0083] The mean durations or holding times, hg, of flows of the service classes are as follows: h s = 1 , 1 , 1 , 4, 4, 6.67, 
where the unit of time is 3 minutes. Thus video flows last on average for 20 minutes. 

[0084] We next describe the aggregate bandwidths offered to streams (s, o), that are also stipulated in the SLA 
and used in the design. We define the matrices U s = {Ly, s = 1 ,2,... ,6, and, furthermore, for compactness we define 
a single base matrix L/from which we obtain U s = k s U t where k s is a scalar multiplier. The multipliers are k s = 0.39, 
0.1 4, 0.1 2, 0.14, 0.1 1 , 0.1 0. The total offered traffic for the real time services (s = 1 and 6) are approximately balanced 
by that for data services. Table I gives the matrix U. 

[0085] The conversion from earned flows to revenue is calculated on the basis that 1 6 Kbps bandwidth carried for a 
unit of time generates unit revenue. 

[0086] The design for the case study was done by the techniques described in D. Mitra et al., "ATM network design 
and optimization: A muftirate loss network framework," IEEE/ACM Trans. Networking 4 531-543 (1996). The design 
gives the flow acceptance ratios for individual streams that exceed 0.99. 

[0087] We considered three scenarios, each with a distinctive traffic pattern that is characterized by the set of actual 
offered aggregate traffic for all streams (s, a), i.e. for all service classes and ingress-egress node pairs. The traffic 
patterns are: 

(i) NORMAL: The ideal case where the offered traffic U {s<3) is identical to the stipulated quantities in the SLA and 
Design. 

(ii) BALANCED ABNORMAL: Half the node pairs, which are selected arbitrarily, have no offered traffic at all, while 
the other half have offered traffic for each of the service classes which are twice the SLA/Design values. 

(Hi) UNBALANCED ABNORMAL: 25% of all node pairs, which are selected arbitrarily, have actual offered traffic 
for each of the service classes which are twice as much as their respective values in the SLA/Design, while for 
the remaining 75% the actual offered traffic is as expected. 

[0088] The lifetimes or holding times of the flows are assumed to be exponentially distributed. 
[0089] Whereas net revenue, W_nef(».), and penalty -■) have been defined above to be cumulative, the results 
presented in this section are for unit time, i.e., obtained from the cumulative quantities by dividing by the length of the 
simulated time. 

[0090] The sample path (time and profile of every flow request) was identically reproduced for all the trials in a given 
scenario. For every trial, 10 million flows are simulated. The statistics reported here are based on results collected 
after a transient period chosen to be sufficiently large for steady state to be reached. The number of flows that contribute 
to the statistics is sufficientfy large to make the confidence intervals negligibly small. 

[0091] The parameters of interest in this study are p, the compensation parameter in the Design/SLA interface; a 
and x, the exponential smoothing parameter and window length in the measurement process, and, importantly, R, the 
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band-width protection parameter 

[0092] The measurement parameters have been chosen empirically. A larger a implies greater smoothing, just as a 
larger window length does. Increasing either one improves the quality of the measurement but at the cost of a slower 
response to significant traffic fluctuations. In our studies, we have found that a satisfying compromise is to set x equal 
to unity, the order of the average holding time, and to have a of 0.8. Also, for the results reported here we have taken 
the smoothing parameter and window length in the SLA monitoring process to be the same as above. 
[0093] Effect of the Bandwidth Protection. The effect of the bandwidth protection on the net revenue is indicated in 
Tablesll, III and IV for normal, balanced abnormal and unbalanced abnormal scenarios, respectively. For these studies, 
we fixed the parameters y and p to 0.5. Here we do not apply the selective crankbacks and recovery-time mechanisms! 
[0094] For normal traffic conditions, the effect of the bandwidth protection and the penalty multiplier on the net revenue 
was found to be small. This is expected because the routing algorithm is optimized specifically for this traffic condition 
so as to maximize the revenue, and also the SLA has been crafted so that the actual carried bandwidth is very close 
to the offered bandwidth, indicating a small loss ratio. As a consequence, the penalty is insignificant in comparison to 
the total generated revenue. Moreover, the generated total revenue decreases slightly as we increase the bandwidth 
protection. This behavior indicates that bandwidth protection is being applied even in the normal condition because of 
the bursty nature of the offered traffic. 

[0095] Turning next to the balanced abnormal traffic pattern, for the first time we observe a noticeable gap between 
the offered bandwidth and the actual carried bandwidth, even though the total offered bandwidth is dose to normal. 
Now most important is the effect of the bandwidth protection; while the protection does not induce a dramatic loss in 
terms of total generated revenue, the penalty is reduced by one order of magnitude when one unit of bandwidth pro- 
tection is applied and by another half when two units of bandwidth protection are applied. In the case of unbalanced 
abnormal traffic, this behavior is accentuated, and in both scenarios we see that a small protection is surprisingly 
beneficial and sufficient. Depending on the penalty multiplier used, our results indicate that here, an optimal value for 
the bandwidth protection parameter is either 1 or 2. 

10096] Effect of Compensation Parameter in Design-SLA Interface. Table V illustrates the effect of varying p for the 
three scenarios when the bandwidth protection parameter R= 1 , the other parameters being the same as above. 



TABLE I 



Base Matrix U, in Mbps 




14.1 


16.5 


2.4 


21.2 


11.8 


4.7 


7.1 


16.5 




56.6 


7.1 


73.1 


35.4 


14.1 


21.2 


18.9 


58.9 




9.4 


87.2 


42.4 


16.6 


25.9 


2.4 


7.1 


7.1 




9.4 


4.7 


2.4 


2.4 


18.9 


70.7 


84.9 


9.4 




54.2 


18.7 


30.7 


11.8 


33.0 


37.7 


4.7 


49.5 




9.4 


14.1 


4.7 


11.8 


14.1 


2.4 


18.9 


9.4 




4.7 


7.1 


18.9 


23.4 


2.4 


28.3 


14.1 


4.7 





TABLE II 



NORMAL Traffic Scenario 


R, Bandwth Prtetn 


Revenue Per unit time 
(x 104) 


Penalty Per unit time 
M s =1 

(X104) 


Net Revenue per unit time 
(X 10*) 


/n^=1 


rrtg=5 


m s = 10 


0 


7.46024 


0.00775 


7.452 


7.421 


7.383 


1 


7.45086 


0.00585 


7.445 


7.422 


7.392 


2 


7.44299 


0.00616 


7.437 


7.412 


7.381 


3 


7.34379 


0.00656 


7.428 


7.402 


7.369 
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TABLE III 



BALANCED ABNORMAL Traffic Scenario 


Ft, Bandwth Prtctn 


Revenue Per unit time 
(x 10*) 


Penalty Per unit time 
M s =1 
(x 104) 


Net Revenue per unit time 
(x 10* 


m^=1 


/7?s=5 


m s = 10 


0 


6.97299 


051680 


6.756 


5.889 


4.805 


1 


6.87995 


0.01519 


6.865 


6.804 


6.728 


2 


6.87025 


0.00248 


6.868 


6.858 


6.845 


3 


6.86073 


0.00486 


6.856 


6.836 


6.812 
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TABLE IV 



UNBALANCED ABNORMAL Traffic Scenario 


R, Bandwth Prtctn 


Revenue Per unit time 
(x 10 4 ) 


Penalty Per unit time 
(X 10 4 ) 


Net Revenue per unit time 
(x 10 4 


m^=1 


mg=S 


m s = 10 


0 


8.43146 


0.65547 


7.776 


5.154 


1.877 


1 


8.28662 


0.09143 


8.195 


7.829 


7.372 


2 


8.22821 


0.03907 


8.189 


8.033 


7.838 


3 


8.19727 


0.04961 


8.148 


7.949 


7.701 



30 TABLE V 



Effect of p For Each Traffic Scenario 


Traffic Scenario 


P 


Revenue 


Penalty 






(X104) 


M s =1 








(X 104) 


Normal 


0.0 


7.44299 


0.00924 




0.5 


7.44299 


0.00616 


Balanced Abnormal 


0.0 


6.87025 


0.00710 




0.5 


6.87025 


0.00248 


Unbalanced Abnormal 


0.0 


8.22821 


0.07051 




0.5 


8.22821 


0.03907 



45 Claims 

1 . A method for computing billing for at least one class of service provided by carrying offered bandwidth between 
source nodes and destination nodes of a communication network, each source-destination node pair sigma in 
association with a service class s to be referred to as a stream (s, o ), the method comprising: 

50 

(a) with respect to at least one stream (s, o ), determining (5, 10,15) for each of two or more consecutive time 
windows whether the network is compliant or non-compliant; 

(b) for each of said time windows, accruing (30) a positive revenue increment (25) for each unit of offered 
bandwidth that is carried; and 

55 (c) for each of said time windows, accruing (45, 50, 60) a negative revenue increment (40) for each unit of 

offered bandwidth that is lost while the network is noncompliant, wherein: 

(A) for measured values of the offered stream bandwidth that do not violate a contracted maximum limit, 
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the network is compliant if (1 0) a measured ratio of carried-to-offered stream bandwidth is at least a con- 
tracted value thereof, but otherwise the network is non-compliant; and 

(B) for measured values of the offered stream bandwidth that violate the contracted maximum limit, the 
network is compliant if (15) a measured value of carried stream bandwidth is at least a contracted value 
thereof, but otherwise the network is non-compliant. 

A method for operating a packetized communication network comprising nodes interconnected by links and sup- 
porting one or more classes of service, wherein, for each pair of nodes of the network consisting of a source node 
and a destination node, each service class has a respective set of permissible routes through the network from 
the source node to the destination node, said service-specific routes to be referred to as service routes; 
the method comprising, for at least one service class s: 

(a) at least at one node, to be referred to as an ingress node, measuring the load imposed by at least some 
service routes for which such node is the source node; 

(b) at the ingress node, comparing (85) the load on at least some service routes with predetermined design 
loads, thereby to determine whether the loading status of such service routes is undersubscribed (90) or over- 
subscribed (95); 

(c) at the ingress node, measuring the load offered to the network by each of a plurality of incoming calls, and 
measuring the load carried by the network in response to said calls; CHARACTERIZED BY: 

(d) at the ingress node, determining (5. 10, 15), from the measurements of offered and earned load, whether 
the network is compliant or non-compliant with a condition; 

(e) at the ingress node, routing at least one incoming call to a destination node according to a procedure (140, 
145, 150, 155, 160, 165) that preferentially selects undersubscribed service routes; 

(f) at the ingress node, in at least one time window, accruing (30) a positive revenue increment (25) for load 
carried by the network due to the routed incoming call; and 

(g) at the ingress node, in at least one time window, accruing (45, 50, 60) a negative revenue increment (40) 
for load offered to the network for routing from the ingress node to the destination node, but lost from the 
network while non-compliant with the condition. 
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FIG. 2 
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FIG. 3 
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FIG. 5 
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FIG. 6 
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